Ford Disclosure No. 200-1453 

SYSTEM AND METHOD FOR VIRTUAL INTERACTIVE DESIGN AND 
EVALUATION AND MANIPULATION OF VEHICLE MECHANISMS 

5 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention relates generally to vehicles 
10 and, more specifically, to a system and method for virtual 
interactive design and evaluation and manipulation of vehicle 
mechanisms of a vehicle. 

2 . Description of the Related Art 

15 Vehicle design, and in particular the design of an 

automotive vehicle, has advanced to a state in which computer- 
assisted design techniques are frequently incorporated in the 
development of a new vehicle or redesign of an existing 
vehicle. Enhanced visualization software tools allow for 

20 interactive display and manipulation of large geometric 
digital models, including digital models developed using 
computer-assisted design (CAD) . In the field of vehicle 
design, the use of computer-assisted design and visualization 
techniques are especially beneficial in designing, packaging, 

25 and assembling various systems incorporated within the vehicle 
to maximize the design and functional capabilities of these 
vehicles. Advantageously, potential vehicle system designs 
can be considered in a timely and cost-effective manner by 
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analyzing a digital representation of a proposed design versus 
preparing an actual physical model. 

One aspect of the design task for a vehicle is the 
coexistence of a digital model and a physical model. An 
5 advantage of the physical model is that it provides a vehicle 
representation of the design that can be evaluated visually 
M: and by touch. 

Q Recently, designers have utilized virtual reality 

III simulation techniques to provide a bridge between the physical 

y s 10 model and the digital model. Virtual reality allows a user to 

interact with a virtual environment, including a virtual 
object in the virtual environment, as if the user was actually 
fT immersed in the virtual environment. 

jrj A user-friendly vehicle device, such as a haptic 

15 device, serves as an interface between the user and the 
virtual environment. An example of a haptic device is 
disclosed in U.S. Patent No. 5,694,013 to Stewart et al., 
entitled "Force Feedback Haptic Interface for a Three- 
Dimensional CAD Surface", the disclosure of which is 
20 incorporated by reference. 

Previous virtual reality interfaces suffer from two 
major disadvantages. The first disadvantage is the poor 
quality of graphic rendering, in terms of resolution, refresh, 
and fidelity. The second disadvantage is the user interface 
25 during exploration and manipulation of the digital model. As 
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a result, it is desirable to provide a system for virtual 
prototyping vehicle mechanisms and allow designers to 
interactively create, modify, and evaluate kinematic and 
dynamic properties of moving parts of a vehicle. It is also 
5 desirable to provide the system with a novel user-interface 
for three-dimensional virtual applications. Thus, there is a 
need in the art for a system and method to design and evaluate 
vehicle mechanisms. 

10 SUMMARY OF THE INVENTION 

Accordingly, the present invention is a system for 
virtual interactive design and evaluation and manipulation of 
vehicle mechanisms using a haptic-user interface. The system 
includes a computer system having a memory, a processor, a 

15 user input device and a display device and a computer 
generated digital model of a vehicle mechanism stored in the 
memory of the computer system. The system further includes a 
haptic-user interface operatively in communication with the 
computer system. The haptic-user interface includes a haptic 

20 end effector device for transmitting force and tactile 
information between a user and the digital model. The user 
interactively creates, modifies, and evaluates kinematic and 
dynamic properties of the vehicle mechanism using the haptic 
end effector device. 



In addition, the present invention is a method for 
virtual interactive design and evaluation and manipulation of 
vehicle mechanisms using a haptic-user interface including the 
steps of importing a digital model of a vehicle mechanism for 
evaluation using a haptic end effector device operatively 
connected to a haptic-user interface. The method also 
includes the steps of determining whether a user is signaling 
a mode change for the digital model and determining the mode 
change if the user is signaling a mode change of the digital 
model. The method includes the steps of getting a position of 
the haptic end effector device into a coordinate reference 
frame for the digital model if the user is not signaling a 
mode change of the digital model. The method further includes 
the steps of determining whether a control handle on the 
digital model is close enough to the haptic end effector 
device and executing the mode change if the control handle is 
close enough. The method includes the steps of applying an 
attraction force to the control handle if the control handle 
is close enough. The method also includes the steps of 
terminating the session if the user desires. 

One advantage of the present invention is that a 
system and method for virtual interactive evaluation and 
manipulation of a digital model of vehicle mechanisms using a 
haptic-user interface is provided. Another advantage of the 
present invention is that the system and method is a computer- 
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user interface for three-dimensional input devices for virtual 
interactive evaluation and manipulation of a digital model of 
vehicle mechanisms. Yet another advantage of the present 
invention is that the system and method allows for virtual 
5 interactive evaluation and manipulation of a digital model of 
vehicle mechanisms. Still another advantage of the present 
H : invention is that the system and method allows designers to 

f~i; 

0 interactively create, modify or edit, and evaluate kinematic 
f|| and dynamic properties of moving parts inside of a vehicle 

10 such as gear-sticks, steering wheels, seats, doors, glove 
7" compartments, and sun-visors with a haptic-user interface for 

fT three-dimensional virtual applications. A further advantage 

1 . L of the present invention is that the system and method employs 

Control-Handles and Spherical-Buttons as an effective 
15 interface between the user and the digital model. Yet a 
further advantage of the present invention is that the system 
and method provides a more natural and effective interaction 
with three-dimensional data. Still a further advantage of the 
present invention is that the system and method is faster, 
20 less expensive, and more flexible than a physical model. 

Other features and advantages of the present 
invention will be readily appreciated, as the same becomes 
better understood, after reading the subsequent description 
taken in conjunction with the accompanying drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a perspective view of a system of 
5 interactive evaluation and manipulation of a digital model of 
vehicle mechanisms, according to the present invention. 

FIG. 2 is a diagrammatic view of the system of 
interactive evaluation and manipulation of a digital model of 
FIG . 1. 

10 FIGS. 3A through 3D are flowcharts of a method for 

virtual interactive evaluation and manipulation of vehicle 
mechanisms using a haptic-user interface, according to the 
present invention,- using the system of FIG. 1. 

FIG. 4 is a diagrammatic view of a control handle 
15 for a mechanism digital model using the system of FIG. 1. 

FIG. 5 is diagrammatic views of a control handle for 
a digital model using the system of FIG. 1. 

FIG. 6 is a graphical representation of a mechanism 
using the system of FIG. 1. 
20 FIG. 7 is a diagrammatic view of a position and 

orientation of a CAD part /assembly of the digital model 
constrained by the mechanism trajectory using the system of 
FIG. 1. 

FIG . 8 is a diagrammatic view of a spherical button 
25 for a digital model using the system of FIG. 1. 



DESCRIPTION OF THE PREFERRED EMBODIMENT (S) 

Referring to the drawings and in particular to 
5 FIGS. 1 and 2, one embodiment of a system 10, according to the 
present invention, for virtual interactive evaluation of 
vehicle mechanisms using a haptic-user interface 12 is 
illustrated graphically. The system 10 includes a haptic-user 
interface 12 operated by a user 14 that controls position, 
10 orientation, and force feedback between the user 14, a 
computer system 16, and a virtual object such as the vehicle 
mechanism . 

The haptic-user interface 12 includes a haptic end 
effector device 18, such as a stylus, pen, or any other 

15 similar gripping device. The user 14 grasps the haptic end 
effector device 18. 

The haptic-user interface 12 also includes a 
hingeable arm 26. In one embodiment, the hingeable arm 2 6 
includes an upper arm 28 and a lower arm 30. The haptic end 

20 effector device 18 is attached to the upper arm 28. The lower 
arm 30 is attached to a base 32. The haptic-user interface 12 
is operatively connected to an actuating device 33, such as a 
servo I/O interface, or other similar device capable of 
transmitting forces to the haptic end effector device 18 

25 through the arms 28,30 in at least three degrees of freedom. 
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Information regarding an interaction with the virtual object 
is transmitted through the haptic end effector device 18 and 
the hingeable arm 2 6 to the computer system 16. It should be 
appreciated that the haptic-user interface 12 provides an 
5 interface between a real world and a virtual world. 

The system 10 further includes the computer system 
?=* 16, as is known in the art, operatively connected to the 

p haptic-user interface 12. The computer system 16 includes a 

fij processor, a controller, and a memory to process information 

y. 10 relevant to the method, according to the present invention, 

for virtual interactive evaluation and manipulation of vehicle 
ft mechanisms. The computer system 16 includes a display device 

; 5 36, such as a video terminal, to display the computer-assisted 

St vehicle design. 

15 The user 14 inputs information into the computer 

system 16 when prompted to do so. Selection and control of 
the information within a screen can be achieved by the user 14 
via a user interactive device 38 such as a keyboard. The set 
of parameters or the set of instructions may be specific to 
20 the method for virtual interactive evaluation and manipulation 
of the vehicle mechanisms, wherein other data and information 
non-specific to the method may already be stored in the memory 
of the computer system 16. One example of an input method is 
a pop-up dialog box containing available information or 
25 instructions. For example, information may be representative 



of different vehicle design alternatives. The computer system 
16 utilizes the set of information or instructions from the 
user 14 and any other information in carrying out the method, 
according to the present invention and discussed in detail 
subsequently, for virtual interactive evaluation and 
manipulation of a digital model 24 of the virtual object such 
as a vehicle mechanism. 

The system 10 also includes a virtual reality 
display system 40. The virtual reality display system 40 
includes a head mounted display device 4 2 as is known in the 
art. The virtual reality display device 42 is worn by the 
user 14 and allows the user 14 to "see" a virtual environment. 
The virtual reality display system 40 is in communication with 
the computer system 16 and provides the user 14 a view through 
a virtual human's eyes or a first person view of the virtual 
environment. For example, the position of the haptic end 
effector device 18 is measured, mapped into a reference frame 
of the digital model, and visually rendered on either the 
display device 36, the head mounted display device 42, or 
another type of graphical display device (not shown) as is 
known in the art. It should be appreciated that the system 10 
can be utilized to evaluate a vehicle mechanism based on 
various factors, such as assembly verification and ergonomics. 

In operation, a controller sends the computer system 
16 a signal representing a joint position for the joints 
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between the haptic end effector device 18, upper arm 28 , lower 
arm 30, and base 32, respectively. The computer system 16 
sends force signals to the controller, which controls the 
servo I/O interface 33 to generate the necessary forces 
5 applied by the haptic end effector device 18. The user 14 can 
view the digital model 24, such as on the display device 36 or 
iU in the head mounted display device 42. 

2l Referring to FIGS. 3A through 3D, a method, 

fb according to the present invention, for virtual interactive 

fl 10 evaluation and manipulation of vehicle mechanisms using the 

haptic-user interface 12 is illustrated. It should be 

s 

appreciated that a parallel process operating on the computer 
system 16 may periodically display the digital model 24 and a 
y simplified representation of the haptic end effector device 18 

15 onto the two-dimensional display device 36 or the head mounted 
display mechanism 42. 

Referring to FIG. 3A, the methodology begins in 
bubble 100 and continues to block 102. In block 102, the user 
14 selects a CAD or tessellated model 24 of a vehicle 

20 mechanism (not shown) for interactive evaluation and 
manipulation and the digital model 24 is imported into the 
methodology. The user 14 selects the digital model 24 with 
the computer system 16. Preferably, the digital model 24 is a 
computer generated digital model and is maintained in a 

25 computer database in the computer 16. 
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From block 102, the methodology advances to block 
104 and sets a "mode" equal to "evaluate". The methodology 
then advances to diamond 106 to be described. It should be 
appreciated that the methodology can be implemented in an 
5 iterative manner. 

In diamond 106, the methodology determines whether 
h& the user 14 is signaling a mode change. Various types of 

O signals are contemplated, including but not limited to, a 

ffj switch integral with the haptic end effector device 18, a user 

lT io interface menu or icon displayed on the display device 36, or 

pressure on the spherical buttons. If the user 14 is 
signaling a mode change, the methodology advances to diamond 
108 and determines whether the new mode equals "create". If 
~ so, the methodology advances to block 110 and sets the mode 

15 equal to create. If not, the methodology advances to diamond 
112 and determines whether the new mode equals "edit 7 '. If 
not, the methodology advances to block 114 and sets the mode 
equal to evaluate. If so, the methodology advances to block 
116 and sets the mode equal to edit. It should be appreciated 
20 that "mechanisms" to be described are used to associate 
trajectories with a CAD part/assembly in the digital model 24. 

It should also be appreciated that there are three operating 
modes: build/create; edit /configure ; and play/evaluate. 

After blocks 110, 114, and block 116 or if the user 
25 14 is not signaling a mode change in diamond 106, the 
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methodology advances to block 118 and gets the position of the 
haptic end effector device 18. The methodology determines a 
haptic end effector device position within a haptic end 
effector device coordinate system reference frame for use in 
5 determining the haptic end effector device position and 
orientation. Preferably, a workspace reference frame, origin 
(0,0,0) and scale of the digital model 24 is established, so 
that the reference frames of the haptic end effector device 18 
and digital model 24 are coordinated. Preferably, the haptic 

10 end effector device position is established using a tracking 
technique as is known in the art. 

The methodology then advances to block 120 and maps 
the position of the haptic end effector device 18 into CAD 
space in the digital model 24 and visualized by the display 

15 device 36 or the head mounted display device 42. The method 
then advances to block 122. 

In block 122, the methodology finds a closest 
"Control-Handle" (CH) on the digital model 24 to the haptic 
end effector device position, P device- Control-Handles are 

20 characterized by: a specific action associated with it; a CAD 
entity such as a CAD part/assembly or a mechanism of the 
digital model 24; an internal state, active or not active; 
and/or a geometric shape and color. Control-Handles are the 
interface to perform action on CAD parts/assemblies of the 

25 digital model, such as to modify a mechanism position, 
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orientation, and size. For example , Control-Handles shape and 
color could be used as a graphical representation of the 
action associated with it. It should be appreciated that each 
mechanism has Control-Handles to allow the user 14 to edit and 
5 play/evaluate the mechanism. The methodology advances to 
diamond 124. 

For example , a cube-like Control-Handle is used to 
apply motion to the white CAD box in Figure 4 . Initially, the 
white CAD box is located in 1. The user 14 positions the 

10 pointing device represented by the arrow in proximity of the 
Control-Handle represented by PO and activates it pressing a 
button on the pointing device, for example the left mouse 
button on the mouse. Motion on the pointing device is now 
replicated to the Control-Handle and on the white CAD box as 

15 well. After moving to PI, the user 14 releases the button and 
now the CAD box is in 2. A sphere-like Control-Handle is used 
to apply rotation to the white CAD box from 2 to 3. It should 
be appreciated that a cube-like Control-Handle is used to move 
the white CAD box from 1 to 2 and a sphere-like Control-Handle 

20 is used to rotate to the white CAD box from 2 to 3 . 

In another embodiment, a slider is represented in a 
CAD as a box constrained to move on a line for a rectilinear 
mechanism. A black, sphere-like Control-Handle is used to 
apply motion to the slider. If the haptic end effector 18 is 

25 used, the force-feedback is applied to pull the hand 34 of the 
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user 14 toward the Control-Handle as illustrated in Figure 5 
as soon as the haptic end effector device 18 is inside a 
proximity sphere as shown by the dotted line circle. If the 
user 14 pulls the haptic end effector device 18 outside the 
5 proximity circle, force feedback will increase to resist 
motion. If the haptic end effector device 18 exits the 
yi proximity sphere in a direction close to parallel to the 

S slider motion line 2, the Control-Handle will become active 

pi and the slider will move. When the user 14 stops moving the 

Tl 10 haptic end effector device 18 , the pointer will get 

increasingly closer to the Control-Handle, till it enters a 
^ smaller proximity sphere 4, and the Control-Handle is now 

deactivated. To exit the influence area of the Control 
P Handle, the user 14 has to pull the haptic end effector device 

15 18 in a direction orthogonal to the slider motion line 5. 
When the haptic end effector device 18 is outside the 
proximity sphere, force feedback is disabled, and the user 14 
is free to move to a different Control-Handle 6. 

In diamond 124 in FIG. 3A, the methodology 
20 determines whether the Control-Handle on the digital model 24 
is close enough to the haptic end effector device position, 
^device- When the hand 34 of the user 14 is in the proximity of 
a Control-Handle, this might change color and shape and the 
user 14 can click a vehicle button on the display device 36 to 
25 virtually grab the Control-Handle. If the Control-Handle is 
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not close enough, the methodology advances to block 12 6 and 
applies an attraction force to the haptic end effector device 
18. It should be appreciated that, when the hand 34 of the 
user 14 gets close to a Control-Handle, the Mechanism 
5 Simulation Application generates a force field to gently pull 
the hand 34 of the user 14 toward the Control-Handle. It 
should also be appreciated that a reference signal is actually 
Q sent to the haptic end effector device 18 that will generate a 

f| I correspondent force feedback. It should be further 

l7 10 appreciated that, by pulling away from the Control-Handle, the 

user 14 can always break free from this force field. It 
should be appreciated that, while the user 14 is moving 
Control-Handles to move, rotate, and/or reshape CAD 
M parts/assemblies, force-feedback can be used to attract the 

15 Control-Handle to the closest CAD feature, such as the closest 
NURBS control point, closest vertex, closest surface. It 
should also be appreciated that if the user 14 positions the 
Control-Handle sufficiently close to such a feature, and if a 
Snap-On option is enabled, the CAD part /assembly would be 
20 adjusted to snap on the selected feature. 

From block 12 6, the methodology then advances to 
block 128 and updates the graphics of the digital model 24 on 
the display device 36 or in the head mounted display device 
42, The method advances to diamond 130 and determines whether 
25 the user 14 is terminating the session by receiving a signal. 
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Various types of signals are contemplated, including but not 
limited to, a switch integral with the haptic end effector 
device 18, a user interface menu or icon displayed on the 
display device 36, pressure on a spherical button, or 
5 orientation of the haptic end effector device 18. If the user 
14 is not terminating the session, the methodology advances to 
iji diamond 106 previously described. If the user 14 is 

*4 terminating the session, the methodology advances to bubble 

'£l 132 and ends or exits the method. 

J; io In diamond 124, if the control-handle on the digital 

mode 24 is close enough to the haptic end effector device 
^ position, the methodology advances to diamond 134 and 

^ determines whether the user 14 is signaling intent to activate 

□ the Control-Handle. If the user 14 is not signaling intent to 

15 activate the Control-Handle, the methodology advances to block 
128 previously described. If the user 14 is signaling intent 
to activate the Control-Handle, the methodology advances to 
diamond 136 and determines whether the mode is equal to 
create. If so, the methodology advances to block 138 and 
20 executes or performs a subroutine called "'create mech" by 
passing through to block 140 in FIG. 3B. It should be 
appreciated that when the hand 34 of the user 14 is in the 
proximity of a Control-Handle, this might change color and 
shape and the user 14 can click a vehicle button the display 
25 device 36 to virtually grab the Control-Handle. It should 
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also be appreciated that, in the create mode, a new mechanism 
is created, shaped, and associated with a CAD part/assembly. 
It should be further appreciated that, in the create mode, the 
user 14 associates a mechanism trajectory with a CAD 
5 part/assembly of the digital model 24. 

Referring to FIG. 4B, in block 140, the methodology 
gets the position of the haptic end effecter device 18 as 
previously described. The methodology then advances to block 
142 and maps the position into CAD space as previously 

10 described. The methodology advances to block 144 and uses the 
position of the haptic end effector device 18 to move the 
trajectory of the mechanism relative to the digital model 24. 

In one embodiment, a curvilinear mechanism M c is 
defined as a 6-pla as illustrated in FIG. 6 and includes: a 

15 reference point Co, relative to the world reference frame; an 
orientation vector n G St 3 , relative to the world reference 
frame; scale vector s 6 9t 3 , relative to the world reference 
frame; a trajectory t (a) , relative to the mechanism reference 
frame; the mechanism handle, alias the current position P c , 

20 and its relative parametric a c that satisfy t (a c ) = P c ; an 
optional dynamics profile, a force f function of the position 
and velocity of the mechanism handle along the given 
trajectory: 

/jt(a c ; dt(oc c ) 
dot c 

25 
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A graphical representation of a mechanism is illustrated in 
FIG. 6. 

In one example, m(t), a function of time, is a valid 

motion of a mechanism M, if for every time t there exist an a 

5 so that m(t) = t (a) . Motion of vehicle components is achieved 

by associating a curvilinear mechanism M ci to a CAD part or 

assembly Ai, so that at any given point P 0 = t(ao), the cross 

product n A {a G )x dt (a 0 )=n rel is known. The current position of 

dot 

the assembly is equal to the current position of the mechanism 

10 P A = P c = t(cc c ). The orientation of the assembly n A (a c ) when 

its position is P A = P c = t (o£ c ) * satisfies the equation: 

n A {ot c )x dt (oig) = n A {a Q )x dt (a 0 ) = n rel 
da da 

15 The CAD part/assembly position and orientation are constrained 

by the mechanism trajectory as illustrated in FIG. 7. 

Further, the definition of a curvilinear mechanism can be 

generalized to a surface mechanism M s , if the motion 

trajectory t (a) is replaced with a motion surface S(u,v)and 

20 introducing a motion domain {u,v}e D c 3t 2 . A multi-DoF 

mechanism, if the motion trajectory t (a) is replaced with a 
motion n-DoF surface S (ui, u 2 , ...u n ) and introducing a motion 
domain {ui, u 2 , ...u n } 6 D c St 2 . in most cases, mechanisms of 
interest have either linear trajectories, such as the front - 
25 rear motion of the seat of the vehicle, or circular-elliptical 



trajectories, such as doors, steering wheel, automatic 
transmission gear-sticks. For example, the handle of manual 
transmission gear-sticks describe elliptical circles/arcs on a 
sphere-ellipsoid. 

From block 144 in FIB. 3B, the methodology then 
advances to block 14 6 and updates the parameters of the haptic 
end effecter device 18 as described above. The methodology 
advances to block 148 and commands force feedback, if needed, 
as previously described. 

From block 148, the methodology advances to block 
150 and updates the graphics of the digital model 24 on the 
display device 36 and the head mounted display device 42 as 
previously described. The methodology advances to diamond 152 
and determines whether the user 14 is signaling end of create. 
Various types of signals are contemplated, including but not 
limited to, a switch integral with the haptic end effector 
device 18, a user interface menu or icon displayed on the 
display device 36, pressure on a spherical button, or 
orientation of the haptic end effector device 18. If the user 
is not signaling end of create, the methodology advances to 
block 140 previously described. If the user 14 is signaling 
end of create, the method advances to bubble 154 and returns 
through block 138 in FIG. 3A to block 128 previously 
described. 
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Referring to FIG. 3A, in diamond 136, if the mode is 
not equal to create, the methodology advances to diamond 156 
and determines whether the mode is equal to edit. If the mode 
is equal to edit, the methodology advances to block 158 and 
5 executes or performs a subroutine called XA edit mech" by 
passing through to block 160 in FIG. 3C. It should be 
appreciated that, when in the edit mode, the user 14 can 
rj translate, rotate, shape, and scale the mechanism trajectory 

ffj and associated CAD assembly. 

H io Referring to FIG. 3C, in block 160, the methodology 

gets the position of the haptic end effecter device 18 as 

y T previously described. The methodology then advances to block 

162 and maps the position into CAD space as previously 

y described. The methodology advances to block 164 and uses the 

i y 

15 position of the haptic end effecter device 18 to move the 
trajectory of the mechanism and CAD assembly of the digital 
model 24 as previously described. The methodology then 
advances to block 166 and updates the parameters of the haptic 
end effecter device 18 as previously described. The 

20 methodology advances to block 168 and commands force feedback, 
if needed, as previously described. The methodology advances 
to block 170 and updates the graphics of the digital model 24 
on the display device 36 and head mounted display device 42 as 
previously described. The methodology advances to diamond 172 

25 and determines whether the user 14 is signaling end of edit. 



21 



Various types of signals are contemplated , including but not 
limited to, a switch integral with the haptic end effector 
device 18, a user interface menu or icon displayed on the 
display device 36, or orientation of the haptic end effector 

5 device 18. If the user 14 is not signaling the end of edit, 
the methodology advances to block 160 previously described. 
If the user 14 is signaling the end of edit, the methodology 
advances to bubble 174 and returns through block 158 in FIG. 
3A to block 128 previously described. 

10 Referring to FIG. 3A, in diamond 156, if the mode is 

not equal to edit, the methodology advances to diamond 17 6 and 
determines whether the mode is equal to evaluate. If the mode 
is not equal to evaluate the methodology advances to bubble 
132 and exits. If the mode is equal to evaluate, the 

15 methodology advances to block 178 and executes or performs a 
subroutine called "evaluate mech" by passing through to block 
180 in FIG. 3D. It should be appreciated that, when in the 
evaluate or play mode, the user 14 can evaluate the CAD 
assembly motion along the mechanism trajectory. It should 

20 also be appreciated that, if, for example, the user 14 is 
evaluating the mechanism motion, the application will move the 
CAD assembly associated with the mechanism in the graphic 
simulation to follow the hand motion of the user 14, and at 
the same time, it will apply the appropriate force feedback to 
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constrain the hand 34 of the user 14 to follow the mechanism 
trajectory. 

Referring to FIG. 3D, in block 180, the methodology 
gets the position of the haptic end effecter device 18 as 
5 previously described. The methodology then advances to block 
182 and maps the position into CAD space as previously 
described. The methodology advances to block 184 and gets the 
force of the user 14 on the haptic end effecter device 18 as 
previously described. From block 184, the methodology 

10 advances to block 186 and maps the force into CAD space in the 
digital model 24. 

From block 186, the methodology advances to block 
188 and finds the closest point on the mechanism trajectory 
relative to the digital model 24 as previously described. The 

15 methodology then advances to block 190 and maps the user force 
into CAD space in the digital model 24. The methodology 
advances to block 192 and adds the virtual forces together. 

The methodology advances to block 194 and calculates 
force/position to control the haptic end effecter device 18 to 

20 replicate physics of real mechanism. The methodology advances 
to block 196 and calculates force/velocity to control the 
haptic end effecter device 18 to replicate physics of real 
mechanism. The methodology advances to block 198 and sends 
force/velocity command to the controller for the haptic end 

25 effecter device 18. 
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After block 198, the methodology advances to block 
200 and uses the position of the haptic end effector device 18 
to update the mechanism position. From block 200, the 
methodology advances to block 202 and updates the graphics of 
5 the digital model 24 on the display device 36 and head display 
device 42 as previously described. The methodology advances 
to diamond 204 and determines whether the user 14 is signaling 
end of evaluate. Various types of signals are contemplated, 
including but not limited to, a switch integral with the 

10 haptic end effector device 18, a user interface menu or icon 
displayed on the display device 36, a spherical button, or 
orientation of the haptic end effector device 18. If the user 
14 is not signaling end of evaluate, the methodology advances 
to block 180 previously described. If the user 14 is 

15 signaling end of evaluate, the method advances to bubble 206 
and returns through block 178 in FIG. 3A to block 128 
previously described. 

In another embodiment, Control-Handles do not scale 
to account for a complex user interface where multiple actions 

20 with even more numerous variant can be taken. A Spherical- 
Button is the basic element of a more complex menu interface, 
the Web-Menu exactly like a click button is the basic element 
of a pop-up menu in Windows95. Its position can be fixed in 
space, such as around a Control-Handle or in a specific 

25 position on the desktop environment, or it could be invoked in 
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real-time when the user 14 clicks a button on the haptic end 
effector device 18 in free space. Essentially a Spherical- 
Button is a sphere-like button that has to be clicked inside- 
out. Force-feedback is applied to pull the haptic end 
5 effector device 18 toward the center of the Spherical-Button 
as illustrated in FIG. 8. 

The Spherical-Button is split into a user-defined 
number of equally sized sectors. A different action is 
associated with each sector. The set of action can be context 

10 dependent, or in other words, different action will be enabled 
depending on the past manipulation history. Different text 
messages may be used, one for each sector, to let the user 14 
know the different actions he/she might perform. Text is 
highlighted, corresponding to the sector the haptic end 

15 effector device 18 is currently in. To click a Spherical- 
Button sector, the user 14 tries to pull away from the button 
from inside the desired sector. At the beginning, increasing 
force-feedback resists outward motion. Then, passed a 
threshold distance, the force decreases , signaling the user 14 

20 that the button is now active. If the user moves back into 
the sphere, the Spherical-Button is clicked. If the user 14 
moves instead the pointer further outside, force-feedback 
increases again, and then drops to zero. No action is taken 
and the Spherical-Button disappears. 
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In yet another embodiment, as soon as the user 14 
passes the initial threshold, clicking can be performed by 
pressing the haptic end effector device 18 instead of 
returning to the center of the sphere. For example, in a 
5 simulation of a gear-stick of a vehicle, a Control-Handle is 
located at the tip of the stick. By default, this Control- 
Handle is used to rotate the stick around its axis. By 
clicking on a Spherical-Button around the Control-Handle, the 
user 14 can switch to a different mode, in which for example, 

10 he/she could move the entire gear-stick assembly to a new 
position, or to a mode in which to edit the range of motion. 

In still another embodiment, a Web Menu architecture 
has multiple spherical buttons organized in a grid structure. 
Force-feedback Snap-On to the lines connecting different 

15 Spherical-Buttons making up the grid speeds up navigation 
from/to different spherical buttons. Each spherical button 
may also be expanded in child sub-spherical buttons, allowing 
for hierarchical organization of information. 

The present invention has been described in an 

20 illustrative manner. It is to be understood that the 
terminology, which has been used, is intended to be in the 
nature of words of description rather than of limitation. 

Many modifications and variations of the present 
invention are possible in light of the above teachings. 

25 Therefore, within the scope of the appended claims, the 



